SLIP: Self-supervision Meets Language-Image Pre-training

Recent work has shown that self-supervised pre-training leads to improvements over supervised learning on challenging visual recognition tasks. CLIP, an exciting new approach with language supervision, demonstrates promising performance a wide variety of benchmarks. In this work, we explore whether can aid in the use supervision for representation Vision Transformers. We introduce SLIP, multi-task framework combining and CLIP pre-training. After pre-training, thoroughly evaluate quality compare both under three distinct settings: zero-shot transfer, linear classification, end-to-end finetuning. Across ImageNet battery additional datasets, find SLIP improves accuracy by large margin. validate our results further experiments different model sizes, training schedules, datasets. Our findings show enjoys best worlds: better than self-supervision (+8.1% accuracy) (+5.2% accuracy). code is available at: github.com/facebookresearch/SLIP .